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Abstract 

SIPRNET the  Secret  Internet  Protocol  Routing  Network,  is  DoIPs  wide 
area  network  for  encrypted  hosts  handling  classified  traffic.  It  is 
managed  by  the  Defense  Information  Systems  Agency.  In  1997  the  U.S. 
Naval  Observatory  established  network  time  synchronization  service  on 
SIPRNET,  with  dedicated  servers  providing  Network  Time  Protocol  (NTP). 
These  servers  are  synchronized  to  the  USNO  Master  Clocks.  An  overview 
of  time  server  design  is  presented,  with  details  on  expected  accuracy  and 
accessibility. 


Introduction 

At  the  26th  meeting  of  PTTI  in  December,  1993,  USNO  reported  on  the 
establishment  of  two  Internet  stratum  1  Network  Time  Protocol  (NTP)  servers. 
Synchronized  to  the  USNO  Master  Clocks  in  Washington,  DC,  these  servers  were 
at  that  time  handling  155,000  NTP  packets  per  month111.  Today  USNO  operates  14 
stratum  1  Internet  NTP  servers,  currently  handling  180  million  packet  requests  per 
month.  These  servers,  at  the  locations  listed  below,  are  Hewlett-Packard  9000/747i 
industrial  workstations  with  embedded  VME  synchronized  generators.  The  sync- 
gens  ( TrueTime  GPS-VME,  VME-SG  and  Datum-Bancomm  bc637vme)  are 
disciplined  by  Master  Clock  IRIG-B  timecode  and  GPS.  The  NTP  servers  provide 
millisecond  synchronization  over  TCP/IP  networks.  Time  transfer  is  performed  by 
exchange  of  local  timestamps  using  User  Datagram  Packets  detailed  in  the 
Internet  Standard  RFC-1305121. 
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Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currendy  valid  OMB  control  number. 
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IP  ADDRESS 
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Time  Service  Dept., 
USNO,  Washington, 

DC 

192.5.41.40 

192.5.41.41 
192.5.41.209 

IRIG 

tick.usno.navy.mil 

tock.usno.navy.mil 

ntp2.usno.navy.mil 

USNO  AMC  Falcon 
AFB,  CO 

204.34.198.40 

204.34.198.41 

IRIG 

tick.usnogps.navy.mil 

tock.usnogps.navy.mil 

Washington  Univ., 

St.  Louis,  MO 

128.252.19.1 

GPS 

ntp-wustl.usno.navy.mil 

Digital  Equipment 
Corp.,  Palo  Alto,  CA 

204.123.2.72 

GPS 

ntp-dec.usno.navy.mil 

MIT,  Cambridge,  MA 

18.145.0.30 

GPS 

ntp-mit.usno.navy.mil 

UCLA,  Los  Angeles, 
CA 

164.67.62.194 

GPS 

ntp-ucla.usno.navy.mil 

Univ.  Houston, 

Houston,  TX 

129.7.1.66 

GPS 

ntp-uh.usno.navy.mil 

Georgia  Institute  of 
Technology,  Atlanta, 
GA 

130.207.244.240 

GPS 

ntp- 

gatech.usno.navy.mil 

Columbia  University, 
New  York,  NY 

128.59.39.48 

GPS 

ntp-cu.usno.navy.mil 

Univ.  Washington, 
Seattle,  WA 

140.142.16.34 

GPS 

ntp-uw.usno.navy.mil 

Ohio  State  University, 
Columbus,  OH 

198.30.92.2 

GPS 

ntp-oar.usno.navy.mil 

SIPRNET,  the  Secret  Internet  Protocol  Routing  Network ,  is  a  Department  of 
Defense  Internet  for  classified  operations.  The  SIPRNET  backbone  is  primarily 
T1  (1.54  Mb/s)  with  about  40  hub  router  sites  in  the  U.S.,  a  dozen  sites  in  Europe, 
and  about  ten  sites  in  Southeast  Asia  and  the  Pacific.  SIPRNET  is  operated  by  the 
Defense  Information  Systems  Agency  (DISA). 

In  response  to  a  number  of  requests  from  SIPRNET  customers,  including  the 
Global  Command  and  Communications  System  (GCCS),  USNO  has  recently  joined 
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with  DISA  in  establishing  NTP  network  time  synchronization  service  on  SIPRNET. 
DISA  has  extended  SIPRNET  hubs  (128Kb  fractional  Tl)  to  the  USNO  Master 
Clocks  at  the  Alternate  Master  Clock  Facility,  Falcon,  CO  (November  97)  and  at 
the  Time  Service  Department  in  Washington,  DC  (December  97).  USNO  has 
provided  NTP  servers  which  are  steered  to  USNO  Master  Clocks. 


USNO  SIPRNET  NTP  SERVERS  1997 


ESS  3 

i3l 
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AMC,  Falcon 

140.49.231.138 

IRIG 

ntp-amc.sipr.smil.mil 

USNO  Washington 

140.49.183.70 

IRIG 

ntp-usno.sipr.smil.mil 

SIPRNET  NTP  SERVER  DESIGN 
The  USNO  SIPRNET  NTP  servers 

are  dedicated  UNIX  workstations  USNO  MASTER  CLOCK  FEEDS 

with  dual  embedded  TmeTime  VME- 
SG  sync-gen  cards.  The  Hewlett- 
Packard  9000/747i  is  an  industrial 
VMEbus  workstation  running  HP- 
UX  10.20.  NTP  release  3-5.90  was 
provided  by  David  Mills,  University 
of  Delaware  (  web 

addre  sshttp  ://w  ww.  eecis.  udel.  edu/~ntp 
/).  A  C-language  NTP  refclock  driver, 
developed  by  the  Time  Service  Dept., 

USNO,  provides  a  memory-mapped 
interface  to  VMEbus  registers  on  the 
TrueTime  VME-SG.  From  these, 

BCD  timecode  can  be  read  with  essentially  zero  latency.  Two  sync-gen  cards 
are  installed  in  the  NTP  server.  Each  card  is  synchronized  to  one  of  USNO 
Master  Clocks  #1  or  #2.  Each  Master  Clock  is  a  Sigma  Tau  Hydrogen  Maser 
providing  5Mhz  to  TST/Trak  6460  Clocks,  which  generate  IRIG-B  timecode 
that  is  distributed  by  TrueTime  560  distribution  amps  with  parallel  output  to 
multiple  NTP  server  inputs. 
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The  NTP  servers  receive  and  transmit  only  unclassified  data,  but  are  located  in  a 
secure  facilities.  The  Washington  connection  to  SIPRNET  uses  a  Larscom 
Access-t  DSU/CSU  and  a  Cisco  AGS+  router.  Between  these  units  an  Allied  Signal 
KTV-7HS  (KG-84)  provides  keyed  encryption. 


Server  timing  characteristics 

The  TrueTime  VME-SG  phase  locks  to  IRIG-B  in  order  to  discipline  its  VCTCXO. 
Phase  shifts  are  internally  applied  in  100  ns  steps131.  Phase  calibration  was 
performed  using  a  time-interval  counter  with  1PPS  reference  from  a  USNO  Master 
Clock.  A  mean  phase  bias  correction  was  then  be  programmed  on  the  VME-SG. 
This  oscillator  is  internally  stable  to  about  200  ns.  Allan  and  time  deviation  plots 
of  the  VME-SG  frequency  and  phase  stability  are  shown  below.  These  are  based 
on  60-second  averages  of  lpps  phase  comparisions  with  a  USNO  Master  Clock. 


TrueTime  VME-SG  Phase  &  Frequency  Stability 


60-sec  1 PPS  averages 
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NTP  software  is  used  to  discipline  the  phase  and  frequency  of  the  UNIX  system 
clock,  which  alone  is  a  poor  oscillator,  showing  drifts  as  much  as  parts  in  10'4. 
The  NTP  driver  software  keeps  the  system  clock  frequency  error  to  generally 
below  2  parts  in  10 7 .  Time  is  buffered  through  system  calls  to  the  UNIX  system 
clock  to  provide  readily  accessible  timestamps  with  a  flywheel  frequency  in  the 
event  of  loss  of  the  sync-gens.  In  addition,  the  server  peers  with  other  NTP 
stratum  1  servers  on  the  network  and  will  maintain  network  synchronization  with 
~20  millisecond  accuracy  in  the  event  of  hardware  clock  failure. 

Internal  frequency  and  time  stability  are  shown  in  the  Allan  deviation  and  time 
deviation  data  shown  below,  which  is  based  on  five  months  of  16-second  NTP 
phase  comparisons  of  a  server's  system  clock  vs.  the  TrueTime  VME-SG  phase. 


USNO  NTP  Server  Stability 


ntp2.usno.navy.mil,  (Jul-Nov  97) 
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Network  Timing  Accuracy 


The  accuracy  of  time  transfer  over  a  packet-switched  Internet,  will  be  considerably 
below  the  internal  timekeeping  ability  of  the  server.  NTP  requires  round-trip 
packet  exchange  between  client  and  server.  The  inherent  asymmetry  of  packet- 
switched  network  paths  contributes  most  of  the  error  in  steering  NTP  clients  to 
servers1  ].  NTP  observes  total  round-trip  network  delay  and  computes  a  dispersion 
which  provides  an  error  bound  for  estimating  clock  offsets.  With  atomic  or  GPS 
NTP  servers  located  at  sites  remote  from  USNO  Washington,  we  can  observe  the 
interplay  between  the  NTP  values  of  network  delay,  dispersion,  and  clock  offset, 
when  the  actual  clock  offset  is  zero.  The  standard  deviation  of  Internet  NTP 
clock  offset  estimations  was  20  ms.  In  previous  tests  on  a  circuit-switched  ISDN 
network  (with  no  round-trip  asymmetries),  the  corresponding  standard  deviation 
was  below  1  mst4J. 

The  expected  accuracy  of  NTP  over  3,000  km  network  paths  is  shown  in  the 
following  histogram  of  NTP  dispersion  between  the  USNO  Washington  and  Falcon 
AFB,  Colorado  SIPRNET  servers. 


SIPRNET  NTP  DISPERSION 
Falcon,  CO  to  Washington,  DC 


NTP  Dispersion  (ms) 
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A  look  at  NTP  clock  offset  estimates  vs.  network  delay  produces  an  NTP  "wedge 
plot,"  as  seen  below.  Points  along  the  edges  of  the  wedge  come  from  busy  network 
routing  queues,  when  round-trip  network  delay  asymmetry  dominates151. 
Unfortunately,  NTP  attributes  half  of  the  delay  asymmetry  to  clock  offset.  By 
dispersing  NTP  servers  widely  across  the  Internet,  we  strive  to  minimize  this  effect 
by  shortening  the  total  synchronization  distance  to  any  client. 


S1PRNET  NTP  OFFSET  VS.  DELAY 

Washington,  DC  -  Falcon,  CO 
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Remote  NTP  Servers 


DISA  and  USNO  plan  to  extend  SIPRNET  NTP  service  to  Europe  and  the  Pacific 
in  1998.  P/Y  code  GPS  receivers  will  provide  accurate  and  reliable  timing. 
USNO's  continual  monitoring  of  GPS  timing  will  assure  that  the  performance  of 
these  GPS  stratum  1  NTP  servers  will  equal  that  of  servers  colocated  at  the 
Master  Clock  sites.  Typical  timing  accuracies  for  NTP  time  service  are 
summarized  below. 


gUfUg  gfl  '  ' 

Accuracy  (peak  to’ peak) 

TrueTime  VME-SG 

0.4  psec 

NTP  server  UNIX  system  clock 

50  psec 

3000  km  circuit-switched  network 

2  ms 

3000  km  packet-switched  network 

20  ms 
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